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Abstract 

Expected coverage and expected length of 90% upper and lower limit and 
68.27% central intervals are plotted as functions of the true signal for various 
values of expected background. Results for several objective priors are shown, 
and formulas for calculation of confidence intervals are obtained. For compar- 
ison, expected coverage and length of frequentist intervals constructed with 
the unified approach of Feldman and Cousins and a simple classical method 
are also shown. It is assumed that the expected background is accurately 
known prior to performing an experiment. The plots of expected coverage 
and length are provided for values of signal and background typical for parti- 
cle experiments with small numbers of events. 
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I. INTRODUCTION 



An intense discussion of various methods for setting confidence limits has led to a number 
of recent publications in the particle physics community. Many such papers discuss methods 
for confidence interval construction in experiments with small numbers of signal events. The 
outcome of such an experiment is modeled by a Poisson statistic 

f(n\s) = e- {s+b \s + b) n /n\ , (1) 

where f(n\s) is the conditional probability of observing n events given the signal s. It is 
assumed in this note that the expected background b is accurately known before performing 
an experiment and can be treated as constant. 

Expected coverage and interval length are important properties of a method. These prop- 
erties were studied in Refs. but there the discussion was mostly focused on Bayesian 
methods with a uniform prior. In this paper I compare expected coverage and interval length 
for several objective Bayesian methods and the unified approach J| of Feldman and Cousins. 

The expected coverage for a specific method is defined as 

oo 

C(s) = £/(n,s)/Hs) ; (2) 

n=0 

J 1 if Si(n) < s < S2(n) ; 
I otherwise. 



I(n,s) 



where [si(n), S2(n)] is the confidence interval constructed for the number of events n with 
this method. The expected length is given by 

oo 

L(s) = Y,(s 2 (n)~s 1 (n))f(n\s) . (3) 

n=0 

For lower limit intervals [s\, +oo) the expected length is infinite, and instead of the length (||) 
an expected distance from zero to the lower limit is plotted: 

oo 

L 1 (s) = Y,Si(n)f(n\s) . (4) 

n=0 

Coverage is mostly a frequentist concept. In a Bayesian analysis one is not concerned 
about coverage as much as about adequately representing one's objective or subjective prior 
belief. This, of course, should not prevent us from investigating properties of expected 
coverage for various Bayesian methods. Another important question is how one should 
define the desired coverage. Due to the discreteness of the Poisson pdf (HD, none of the 
methods provides exact coverage for all values of s. In a frequentist approach one typically 
requires at least minimal coverage for every value of the true signal s, and thus frequentist 
intervals overcover on the average. This overcoverage is not an intrinsic feature of the 
frequentist interval construction but merely a consequence of the empirical conservatism that 
many physicists find desirable. Statistics literature has examples of consistent frequentist 
methods [^,|J that lack this minimal coverage property. One can argue that a method that 
gives the requested mean coverage should be preferred over the one that always overcovers. 
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No matter what the solution to this problem is, further discussion of this issue is beyond 
the scope of this paper. 

While confidence intervals and the expected coverage are invariant under metric trans- 
formation, the expected length and mean coverage are not. A transformation of metric, of 
course, involves a corresponding Jacobian transformation of the prior density in a Bayesian 
approach. For example, if the confidence interval for s is [sx(n), 82(71)], then a confidence 
interval for the quantity 1/s is obtained by a simple inversion: [l/s2(n), l/si(n)], and the 
expected coverage has the same functional dependence upon signal s. But the expected 
interval length and mean coverage apparently change due to the choice of a new metric. 



II. BAYESIAN INTERVALS 

In a Bayesian approach one has to assume a prior probability density function 7r(s) for 
unknown signal and then obtain a posterior pdf 



7T s \n) 



f(n\s)ir(s) 



(5) 



Io° f(n\s)n(s)ds ' 
A confidence interval [s\, S2] is then found by solving equations 

Jo 1 n(s\n)ds ; , 
J£" 7r(s\n)ds ; W 

for si and s 2 . Here, ot\ and a 2 are the probabilities of the left and right tails of the posterior 
pdf, respectively, and 

CL = 1 — a>i — «2 (7) 

is the confidence level of the constructed interval [si,s 2 ]. For example, to compute a 90% 
upper limit, one should put a% = and a 2 = 0.1; for a 90% lower limit these values are a.\ = 
0.1 and a 2 = 0; and to obtain a 68.27% central interval, one should use ot\ = ot 2 = 0.15865. 

In particle physics it has been customary to choose a "non-informative" or "objective" 
prior 7r(s), i.e., a prior that claims absence of any knowledge about the true signal s. The 
three most popular candidates for an objective prior in a measurement without background 
are a flat pdf Jeffreys' prior 1/s |T0|-T2| and the 1/y/s prior []13 |. The last two were 



derived from first principles, and the motivation for a flat prior is merely that it is "obvious" . 
If non-zero background is expected, one can apply the same first principles and obtain 
l/(s + b) and l/\/s + b for the two derived prior pdf's, respectively. One can also argue fTl 



that the 1/y/s prior should be used even if non-zero background is expected because our 
prior knowledge of background should not affect our prior ignorance about signal. The latter 
argument can be applied to the 1/s prior as well, but this prior gives a divergent posterior 
pdf for non-zero background b > and is therefore unacceptable. The discussion below is 
confined to these four prior pdf's: flat, l/(s + b), 1/y/s and 1/y/s + b. 
For a prior distribution 

tt(s) = l/s m ; < m < 1 ; (8) 
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the posterior pdf is given by 



e- s s~ m (s + b) n /n\ 
~ ELo bkT ( n -k-m + 1)/ {k\{n - k)\) ' (9) 



and the confidence interval can be computed using 

i _ fsr^n ifc r(n-fc-m+l,si) \ / I \-^n L k T{n-k-m+l) \ 
1 ^k=0 u _ k\(n-k)\ ) I \l^k=O u k\(n-k)\ ) 

-m-j 



ST n 7k ^(n-k-m+l,s 2 ) \ / (\-^n i k T(n-k-m+l) 
K l^k=O u k\(n-k)\ ) I \2^k=0 u k\(n 



(10) 



where 



p-1 



T(p,fi) = x v ~ x e~ x dx\ p>0; fi>0; (11) 



is an incomplete gamma-function, and T(p) = T{p, 0) is the standard gamma-function. 
For a prior distribution 

tt(s) = l/(s + b) m ; 0<m<l; (12) 

a similar derivation leads to the posterior pdf 

e -( S +fc)( s + & )n- m 

7r(s n = —^7 7^ . (13) 

v 1 ' r(n-m + l,6) 



The confidence interval is then given by 



■i _ r(n-m+l,si+b) 

r(n-m+l,6) ' / u \ 

r(n-m+l,s 2 +b) \ lq: ) 
r(ra-m+l,fe) 



At m = (flat prior) or b = (no background) the posterior distributions @ and ([13|) are, 
of course, identical. The posterior pdf fll3|) is divergent at m = 1 (Jeffreys' prior) and n = 
(no events observed). I assume that this is equivalent to producing an interval of zero length 
and set 1(0, s) in Eqn. (H) to zero. 



III. FREQUENTIST INTERVALS 

For comparison, expected coverage and length of confidence intervals constructed with 
the unified approach 0] of Feldman and Cousins and a simple classical method are also 
plotted. The simple classical approach, to which I will refer below as "standard", assumes 
a confidence interval of the form [0, S2] for upper limit calculation and that of the form 
[si, +00) for a lower limit. Rules for construction of frequentist confidence belts are outlined 



in a famous paper [15] by Neyman. The confidence belt [rii(s) , n 2 (s)] for the standard 

(15) 



classical method is given by 

= smallest integer for which J2k=o f (k\ s ) > a 2 



= largest integer for which Y^T=n 2 f(k\ s ) > a i ■ 

Here, the same convention as in the previous section is used for a\ and a 2 , e.g., to compute 
a 90% upper limit, one should put a>i — and a 2 = 0.1 etc. This belt is then used to 
construct a confidence interval [si(n), S2(n)] for a specific number of events n in accordance 



with Ref. 15 
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IV. DISCUSSION 



Expected coverage and interval length for the four objective priors and two frequentist 
methods are plotted versus true signal in the range < s < 10 with a step 0.001 for 4 
different values of expected background 6 = 0, 1, 3, and 6. The plots are shown in Figs. 1- 
12. The expected length was computed by summing terms in Eqn. (^) from to 100 which 
was enough to achieve a good accuracy for the chosen values of s and b. 

Both frequentist methods produce intervals of at least minimal coverage because this 
conservative requirement was used in construction of their confidence belts. Without back- 
ground the Bayesian method with a flat prior and the standard classical procedure give 
identical upper limit values, and this is also true for the Bayesian with Jeffreys' prior and 
the standard classical methods in case of lower limits. At any background the Bayesian 
method with a flat prior provides at least minimal coverage for upper limit intervals and 
undercovers for lower limits, and this is reversed for Jeffreys' prior. None of the Bayesian 
procedures provides minimal coverage for all values of the true signal and for all types of 
confidence intervals. In terms of coverage, the 1/VsTb prior is the most versatile choice 
among the Bayesian methods. It provides a reasonable mean coverage for all types of confi- 
dence intervals while the 1 / \fs prior tends to undercover for upper limits and overcover for 
lower limits. For central intervals the expected coverage of all Bayesian methods oscillates 
about the required confidence level. The Bayesian method with a flat prior always gives 
longer central and upper limit intervals than that with the 1 / ys + b prior, and intervals 
produced with this prior are in turn longer than those obtained with Jeffreys' l/(s + b) 
prior. The l/\fs prior produces short intervals at small signal values but, as the signal 
increases, these intervals become longer than those produced by the 1/a/s + b and l/(s + b) 
priors. Expected lengths for all Bayesian methods approach each other asymptotically as the 
signal increases. The unified approach produces short 90% intervals at large signal values, 
but one should keep in mind that here the expected length of a two-sided unified interval is 
plotted as opposed to lengths of one-sided intervals for all other methods. 
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FIG. 1. Expected coverage of 90% upper limit (solid) and lower limit (dashed) intervals 
for the expected background 6 = 0. For the unified approach the expected coverage of 90% 
intervals is plotted. 
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FIG. 2. Expected coverage of 68.27% central intervals for the expected background 6 = 0. 
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FIG. 3. Expected coverage of 90% upper limit (solid) and lower limit (dashed) intervals 
for the expected background 6=1. For the unified approach the expected coverage of 90% 
intervals is plotted. 
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FIG. 4. Expected coverage of 68.27% central intervals for the expected background 6=1. 
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FIG. 5. Expected coverage of 90% upper limit (solid) and lower limit (dashed) intervals 
for the expected background 6 = 3. For the unified approach the expected coverage of 90% 
intervals is plotted. 
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FIG. 6. Expected coverage of 68.27% central intervals for the expected background 6 = 3. 
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FIG. 7. Expected coverage of 90% upper limit (solid) and lower limit (dashed) intervals 
for the expected background 6 = 6. For the unified approach the expected coverage of 90% 
intervals is plotted. 
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FIG. 8. Expected coverage of 68.27% central intervals for the expected background 6 = 6. 
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FIG. 9. Expected length of 90% upper limit (upper bunch of curves) and lower limit (lower 
bunch of curves) intervals on the left and expected length of 68.27% central intervals on the right. 
The expected background is b = 0. 
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FIG. 10. Expected length of 90% upper limit (upper bunch of curves) and lower limit (lower 
bunch of curves) intervals on the left and expected length of 68.27% central intervals on the right. 
The expected background is b = 1. 
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FIG. 11. Expected length of 90% upper limit (upper bunch of curves) and lower limit (lower 
bunch of curves) intervals on the left and expected length of 68.27% central intervals on the right. 
The expected background is b = 3. 
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FIG. 12. Expected length of 90% upper limit (upper bunch of curves) and lower limit (lower 
bunch of curves) intervals on the left and expected length of 68.27% central intervals on the right. 
The expected background is b = 6. 
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